Image processing apparatus, image pickup apparatus, image processing method, and non-transitory computer-readable storage medium for improving quality of image

ABSTRACT

An image processing apparatus ( 204 ) includes a generator ( 204   a ) which generates difference information relating to a plurality of parallax images, a gain distribution determiner ( 204   b ) which determines a gain distribution based on the difference information, an intensity determiner ( 204   c ) which determines an unnecessary component intensity based on the gain distribution, and a reducer ( 204   d ) which generates an unnecessary component reduction image in which an unnecessary component is reduced based on the parallax image and the unnecessary component intensity.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing method forimproving a quality of a captured image.

2. Description of the Related Art

In image capturing through an image pickup apparatus such as a camera,part of light incident on an optical system may be reflected by asurface of a lens and a member holding the lens and arrive at an imagingplane as unnecessary light. This unnecessary light appears as anunnecessary component such as a ghost and a flare in a captured image.When a diffractive optical element is used in a telephoto lens tocorrect longitudinal (axial) chromatic aberration and chromaticaberration of magnification, light from a high intensity object such asthe sun outside an angle of view for the image capturing may be incidenton the diffractive optical element, generating unnecessary light as anunnecessary component over the entire image. Previously, a method ofremoving the unnecessary component by using digital image processing isknown.

Japanese Patent Laid-open No. 2008-54206 discloses a method of detectingany ghost based on a difference image indicating a difference between animage (in-focus image) when an optical system is in focus on an objectand an image (defocus image) when the image pickup optical system is outof focus. However, the method disclosed in Japanese Patent Laid-open No.2008-54206 requires image capturing to be performed a plurality of timesand thus is not suitable for still image pickup and moving image pickupof a moving object.

Japanese Patent Laid-open No. 2011-205531 discloses a method ofdetecting a ghost based on comparison of a plurality of parallax imagescaptured by a single-lens stereoscopic image pickup. The methoddisclosed in Japanese Patent Laid-open No. 2011-205531, which obtains aplurality of parallax images by single image capturing, is applicable tostill image pickup and moving image capturing of a moving object.

However, in the method disclosed in Japanese Patent Laid-open No.2011-205531, an optical path for the ghost is displaced from an idealpupil-divided optical path, and accordingly the ghost cannot beeffectively reduced if the same ghost appears in both a main pixel and asubpixel while a luminance distribution is different.

SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus, an imagepickup apparatus, an image processing method, and a non-transitorycomputer-readable storage medium which are capable of effectivelydetermining an intensity of an unnecessary component contained in acaptured image without imaging a plurality of times to reduce theunnecessary component from the captured image.

An image processing apparatus as one aspect of the present inventionincludes a generator configured to generate difference informationrelating to a plurality of parallax images, a gain distributiondeterminer configured to determine a gain distribution based on thedifference information, an intensity determiner configured to determinean unnecessary component intensity based on the gain distribution, and areducer configured to generate an unnecessary component reduction imagein which an unnecessary component is reduced based on the parallax imageand the unnecessary component intensity.

An image pickup apparatus as another aspect of the present inventionincludes an image pickup device configured to photoelectrically convertan optical image formed via an optical system to output a plurality ofparallax images, a determiner configured to determine differenceinformation relating to the plurality of parallax images, a calculatorconfigured to calculate a gain distribution based on the differenceinformation, an intensity determiner configured to determine anunnecessary component intensity based on the gain distribution, and areducer configured to generate an unnecessary component reduction imagein which an unnecessary component is reduced based on the parallax imageand the unnecessary component intensity.

An image processing method as another aspect of the present inventionincludes the steps of determining difference information relating to aplurality of parallax images, calculating a gain distribution based onthe difference information, determining an unnecessary componentintensity based on the gain distribution, and generating an unnecessarycomponent reduction image in which an unnecessary component is reducedbased on the parallax image and the unnecessary component intensity.

A non-transitory computer-readable storage medium as another aspect ofthe present invention stores a program causing a computer to execute aprocess including the steps of determining difference informationrelating to a plurality of parallax images, calculating a gaindistribution based on the difference information, determining anunnecessary component intensity based on the gain distribution, andgenerating an unnecessary component reduction image in which anunnecessary component is reduced based on the parallax image and theunnecessary component intensity.

An image processing apparatus as another aspect of the present inventionincludes an unnecessary component determiner configured to generatedifference information relating to a plurality of parallax images todetermine an unnecessary component based on the difference information,a gain distribution determiner configured to determine a gaindistribution based on the unnecessary component, and a reducerconfigured to generate an unnecessary component reduction image in whichan unnecessary component is reduced based on the parallax image, theunnecessary component, and the gain distribution.

An image pickup apparatus as another aspect of the present inventionincludes an image pickup device configured to photoelectrically convertan optical image formed via an optical system to output a plurality ofparallax images, an unnecessary component determiner configured togenerate difference information relating to the plurality of parallaximages to determine an unnecessary component based on the differenceinformation, a gain distribution determiner configured to determine again distribution based on the unnecessary component, and a reducerconfigured to generate an unnecessary component reduction image in whichan unnecessary component is reduced based on the parallax image, theunnecessary component, and the gain distribution.

An image processing method as another aspect of the present inventionincludes the steps of generating difference information relating to aplurality of parallax images to determine an unnecessary component basedon the difference information, determining a gain distribution based onthe unnecessary component, and generating an unnecessary componentreduction image in which an unnecessary component is reduced based onthe parallax image, the unnecessary component, and the gaindistribution.

A non-transitory computer-readable storage medium as another aspect ofthe present invention stores a program causing a computer to execute aprocess including the steps of generating difference informationrelating to a plurality of parallax images to determine an unnecessarycomponent based on the difference information, determining againdistribution based on the unnecessary component, and generating anunnecessary component reduction image in which an unnecessary componentis reduced based on the parallax image, the unnecessary component, andthe gain distribution.

Further features and aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of illustrating a procedure of an image processingmethod in each of Embodiments 1 and 2.

FIG. 2 is an exemplary output image obtained by the image processingmethod in each of Embodiments 1 and 2.

FIG. 3 is a relational diagram of a light-receiving portion of an imagepickup element and a pupil of an optical system in an image pickupsystem in each of Embodiments 1 and 2.

FIG. 4 is a schematic diagram of the image pickup system in each ofEmbodiments 1 and 2.

FIG. 5 is a block diagram of an image pickup apparatus in eachembodiment.

FIGS. 6A to 6C are diagrams of a configuration of the optical system andan explanatory diagram of unnecessary light occurring in the opticalsystem each embodiment.

FIG. 7 is an explanatory diagram of unnecessary light passing through anaperture stop of the optical system in each of Embodiments 1 and 2.

FIG. 8 is exemplary output images obtained by the image processingmethod in each of Embodiments 1 and 2.

FIG. 9 is exemplary output images obtained by the image processingmethod in each of Embodiments 1 and 2.

FIG. 10 is a flowchart of illustrating an image processing method inEmbodiment 1.

FIG. 11 is a flowchart of illustrating an image processing method ineach of Embodiments 2 and 3.

FIG. 12 is a diagram of illustrating a reduction rate distribution inEmbodiment 2.

FIG. 13 is a diagram of illustrating an image pickup element inEmbodiment 3.

FIG. 14 is an explanatory diagram of unnecessary light passing throughan aperture stop of the optical system in Embodiment 3.

FIG. 15 is a diagram of illustrating a procedure of an image processingmethod in Embodiment 3.

FIG. 16 is a diagram of illustrating a procedure of an image processingmethod in Embodiment 3.

FIG. 17 is a diagram of illustrating a reduction rate distribution inEmbodiment 3.

FIG. 18 is exemplary output images obtained by the image processingmethod in Embodiment 3.

FIG. 19 is a diagram of illustrating an image pickup system inEmbodiment 4.

FIG. 20 is a diagram of illustrating an image pickup system inEmbodiment 4.

FIG. 21 is a diagram of illustrating an image pickup system inEmbodiment 4.

FIG. 22 is a diagram of illustrating a conventional image pickupelement.

FIGS. 23A and 23B are diagrams of illustrating images obtained throughthe image pickup system in FIG. 19.

FIG. 24 is a diagram of illustrating images obtained through the imagepickup system in each of FIGS. 20 and 21.

FIG. 25 is a diagram of illustrating an example of an image pickupapparatus in Embodiment 4.

FIG. 26 is a diagram of illustrating an example of an image pickupapparatus in Embodiment 4.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to the accompanied drawings.

In each embodiment, an image pickup apparatus capable of generating aplurality of parallax images includes an image pickup system that guidesa plurality of light beams passing through regions of a pupil of anoptical system (image pickup optical system) that are different fromeach other, to light-receiving portions (pixels) of an image pickupelement that are different from each other and perform photoelectricconversions.

Embodiment 1

First of all, Embodiment 1 of the present invention will be described.FIG. 3 illustrates a relation between light-receiving portions of animage pickup element in an image pickup system in this embodiment and apupil of an optical system. In FIG. 3, symbol ML represents a microlens, and symbol CF represents a color filter. Symbol EXP represents anexit pupil (the pupil) of the optical system, and symbols P1 and P2represent regions of the exit pupil EXP. Symbols G1 and G2 representpixels (light-receiving portions), and one pixel G1 and one pixel G2make a pair (the pixels G1 and G2 are disposed to share a single microlens ML). The image pickup element includes an array of a plurality ofpairs (pixel pairs) of the pixels G1 and G2. The paired pixels G1 and G2have a conjugate relation with the exit pupil EXP via the shared (thatis, provided for each pixel pair) micro lens ML. In each embodiment, thepixels G1 and G2 arrayed in the image pickup element are also referredto as pixel groups G1 and G2, respectively.

FIG. 4 is a schematic diagram of the image pickup system in thisembodiment which is assumed to have a configuration in which instead ofthe micro lens ML illustrated in FIG. 3, a thin lens is provided at theposition of the exit pupil EXP. The pixel G1 receives a light beampassing through a region P1 of the exit pupil EXP. The pixel G2 receivesa light beam passing through a region P2 of the exit pupil EXP. SymbolOSP represents an object point for which image pickup (imaging or imagecapturing) is performed. The object point OSP does not necessarily needto have an object located thereon. A light beam passing through theobject point OSP is incident on one of the pixel G1 and the pixel G2depending on a position (the region P1 or the region P2 in thisembodiment) in the pupil (exit pupil EXP) through which the light beampasses. Travelling of light beams through regions of the pupil that aredifferent from each other corresponds to separation of incident lightfrom the object point OSP by its angle (parallax). In other words, foreach micro lens ML corresponding to the pixels G1 and G2, an image basedon an output signal from the pixel G1 and an image based on an outputsignal from the pixel G2 are generated as a plurality of (in thisexample, a pair of) parallax images having parallaxes with each other.Hereinafter, reception of light beams passing through the regions of thepupil that are different from each other by the light-receiving portions(pixels) different from each other may be referred to as pupil division.

When the conjugate relation is not completely held due to, for example,a position shift of the exit pupil EXP illustrated in FIGS. 3 and 4, orwhen the regions P1 and P2 partially overlap with each other, aplurality of obtained images are still treated as parallax images inthis embodiment. A minimum element that constitutes an image is called apixel (pixel signal), which is distinguished from a pixel on the imagepickup element, and each pixel represents a light intensity and coloraccording to its numerical value. A value of each pixel is referred toas a pixel value. The pixel value is equal to a luminance value when theimage is a monochrome image, and each embodiment in the presentinvention will describe the monochrome image for simplicity. Therefore,in each embodiment, the pixel value and the luminance value have thesame meaning. When the image is an RGB color image, the same calculationcan be performed for each color of the pixel values. This is true alsoin each of the following embodiments.

Next, referring to FIG. 5, an image pickup apparatus that executes animage processing method in this embodiment will be described. FIG. 5 isa block diagram of illustrating a configuration of an image pickupapparatus 200 in this embodiment. An optical system 201 (image pickupoptical system) includes an aperture stop 201 a and a focus lens 201 b,and causes light from an object (not illustrated) to be imaged(condensed) on an image pickup element 202. The image pickup element 202includes a photoelectric conversion element such as a CCD sensor and aCMOS sensor, and receives light beams passing through regions of thepupil that are different from each other, through pixels(light-receiving portions) corresponding to the respective regions(performs the pupil division), as described referring to FIGS. 3 and 4.In this manner, the image pickup element 202 performs a photoelectricconversion on an object image (optical image) formed via the opticalsystem 201 and outputs image signals (analog electric signals) as aplurality of parallax images. An A/D converter 203 converts the analogelectric signals output from the image pickup element 202 into digitalsignals, and then outputs these digital signals to an image processor204.

The image processor 204 performs typical image processing on the digitalsignals, and also performs determination processing of unnecessary light(unnecessary component) and correction processing to reduce or removethe unnecessary light. In this embodiment, the image processor 204corresponds to an image processing apparatus incorporated in the imagepickup apparatus 200. The image processor 204 includes an unnecessarycomponent detector 204 a (generator), a gain distribution acquirer 204 b(gain distribution determiner), an unnecessary component intensitydeterminer 204 c (intensity determiner), and an unnecessary componentreducer 204 d (reducer).

The unnecessary component detector 204 a generates (acquires) parallaximages and detects (determines) an unnecessary component based on theparallax images. The gain distribution acquirer 204 b calculates a gaindistribution in an image to determine an unnecessary component intensityat a subsequent stage. The unnecessary component intensity determiner204 c determines the unnecessary component intensity to be reduced basedon the detected unnecessary component and the gain distribution. Theunnecessary component reducer 204 d reduces an unnecessary componentfrom each parallax image depending on the unnecessary componentintensity. In this embodiment, the parallax images can be output andgenerated as “an image formed only by the pixel group G1” and “an imageformed by only by the pixel group G2” in a form previously separatedinto the two images. Alternatively, “an image formed only the pixelgroup G1” and “a synthesized image of the pixel groups G1 and G2” may beoutput first, and then an image formed only by the pixel group G1 may besubtracted from the synthesized image to calculate and obtain an imagecorresponding to the image formed only by the pixel group G2.

The output image (image data) processed by the image processor 204 isstored in an image recording medium 209 such as a semiconductor memoryand an optical disk. The output image from the image processor 204 canbe displayed on a display unit 205. A storage unit 208 (memory) storesan image processing program and various kinds of information needed forthe image processing by the image processor 204.

A system controller 210 (controller, processor, or CPU) controls theoperation of the image pickup element 202, the processing by the imageprocessing unit 204, and the optical system 201 (the aperture stop 201 aand the focus lens 201 b). An optical system controller 206 performsmechanical drive of the aperture stop 201 a and the focus lens 201 b ofthe optical system 201 in response to a control instruction from thesystem controller 210. The aperture stop 201 a has its opening diametercontrolled in accordance with a set aperture value (F-number). The focuslens 201 b has its position controlled by an autofocus (AF) system and amanual focus mechanism (not illustrated) to perform focusing (focuscontrol) in accordance with an object distance. A state detector 207acquires current image capturing condition information in response to acontrol instruction from the system controller 210. In this embodiment,the optical system 201 is included as part of the image pickup apparatus200 (integrally with the image pickup apparatus 200) including the imagepickup element 202, but is not limited thereto. Like a single-lensreflex camera, the image pickup system may include an interchangeableoptical system (interchangeable lens) detachably attached to an imagepickup apparatus body.

FIGS. 6A to 6C are a configuration diagram of the optical system 201 andexplanatory diagrams of unnecessary light occurring in the opticalsystem 201. FIG. 6A specifically illustrates an exemplary configurationof the optical system 201. In FIG. 6A, symbol STP represents an aperturestop (corresponding to the aperture stop 201 a), and symbol IMGrepresents an imaging plane. The image pickup element 202 illustrated inFIG. 5 is disposed at the position of the imaging plane IMG. FIG. 6Billustrates a case in which strong light from the sun denoted with SUNas an exemplary high luminance object is incident on the optical system201, and light reflected at a surface of a lens included in the opticalsystem 201 arrives as an unnecessary component A (unnecessary light suchas a ghost or a flare) at the imaging plane IMG. FIG. 6C illustrates acase in which strong light is incident similarly to FIG. 6B, and lightreflected at a surface of a lens different from the surface by which theunnecessary component A is reflected arrives as an unnecessary componentB (unnecessary light such as a ghost and a flare) at the imaging planeIMG.

FIG. 7 illustrates the regions P1 and P2 (pupil regions or pupildivision regions) of the aperture stop STP, through which light beamsincident on the pixels G1 and G2 illustrated in FIG. 4 pass. Theaperture stop STP can be assumed to correspond to the exit pupil EXP(i.e., virtual image when seen from an image plane position of theoptical system 201) of the optical system 201, but in practice, it isoften the case that the aperture stop STP and the exit pupil EXP aredifferent from each other. Although a light beam from the high luminanceobject (SUN) passes through an almost entire region of the aperture stopSTP, a region through which the light beams to be incident on the pixelsG1 and G2 pass is divided into the regions P1 and P2 (pupil regions). Inthe example illustrated in FIGS. 6B and 6C, the light beam from the highluminance object passes through a region approximately at the lower halfof the aperture stop STP, and it is a situation in which part of thelight beam passes through the region P1 and the remaining entire lightbeam passes through the region P2 referring to FIG. 4. The light beampassing through the region P1 is incident on the pixel G1, and the lightbeam passing through the region P2 is incident on the pixel G2.

Next, referring to FIGS. 1 and 2, a method of determining an unnecessarycomponent as an image component that appears through a photoelectricconversion of unnecessary light in a captured image generated by theimage pickup apparatus 200 will be described. FIG. 1 is a diagram ofillustrating a procedure of the image processing method in thisembodiment. FIG. 2 is an example of an output image obtained by theimage processing method in this embodiment. When an image is captured byusing the optical system 201 illustrated in FIG. 6A, an unnecessarycomponent A that occurs in the optical path of FIG. 6B and anunnecessary component B that occurs in the optical path of FIG. 6Coverlap with each other. However, in FIGS. 1 and 2, for simplifyingdescriptions, the unnecessary components A and B are illustratedseparately. Regardless of whether a plurality of unnecessary componentsare overlapped or separated, each of an idea and a basic concept of thisembodiment is the same, and each of a method of calculating a gaindistribution and a method of reducing the unnecessary component is alsothe same.

FIG. 2 illustrates a captured image which is generated by “imagingwithout pupil division”. In this captured image, for simplicity, a fineobject is omitted, and a part in gray as a background (including anobject) and two squares (unnecessary components A and B) horizontallydisposed that indicate ghosts (unnecessary components with higherluminances than that of each of the object and the background) appear.In reality, objects are somewhat transparent at the background of theseunnecessary components. The unnecessary component corresponds tounnecessary light on a captured object, and thus has luminance higherthan that of the captured object. Therefore, it is illustrated with ahigher luminance than the gray part corresponding to the background.This is true also in other embodiments described below.

FIG. 1 (A-1) and FIG. 1 (B-1) illustrate a pair of parallax images whichare obtained as a result of the photoelectric conversion of the lightbeams passing through the regions P1 and P2 (pupil regions) by the pixelgroups G1 and G2. A difference (parallax component of an object)corresponding to a parallax of an image component exists between thepair of parallax images. However, for simplifying descriptions, theparallax component is omitted. The pair of parallax images contain theunnecessary components A and B schematically illustrated as whitesquares with uniform luminances, and the luminances are different fromeach other between the parallax images. In this embodiment, as describedabove, the example in which the unnecessary components A and B areseparated without overlapping with each other is illustrated, butinstead, these may overlap with each other to have a luminancedifference. In other words, positions or luminances of the unnecessarycomponent may be different from each other between the parallax images.

FIG. 1 (A-2) and FIG. 1 (B-2) illustrate luminance cross section of thepair of parallax images along the horizontal direction at the vicinityof the center in the vertical direction. Numerical values in graphs ofFIG. (A-2) and FIG. 1 (B-2) are luminance values of the unnecessarycomponents. For example, in FIG. 1 (A-2), a luminance value at thebackground is 70, and both luminance values of the unnecessarycomponents A and B are 130. FIG. 1 (C-1) illustrates an image(synthesized parallax image) obtained by adding and synthesizing theimages of FIG. 1 (A-1) and FIG. 1 (B-1). FIG. 1 (C-2) illustrates aluminance cross section of the synthesized parallax image along thehorizontal direction at the vicinity of the center in the verticaldirection. This synthesized parallax image is equivalent to the capturedimage of FIG. 2 generated by the “imaging without pupil division”. Inthis embodiment, by adding and synthesizing the pair of parallax images,a brightness (luminance) that is equivalent to that of the capturedimage generated by the “imaging without pupil division” is obtained.Instead, an image pickup apparatus which is capable of obtaining abrightness (luminance) that is equivalent to that of the captured imagegenerated by the “imaging without pupil division” by averaging (addingand averaging) the pair of parallax images may be used. This case willbe described in Embodiment 3 below.

FIG. 1 (D-1) illustrates an image obtained by subtracting the image ofFIG. 1 (B-1) from the image of FIG. 1 (A-1) with respect to the pair ofparallax images. FIG. (D-2) illustrates a luminance cross section alongthe horizontal direction at the vicinity of the center in the verticaldirection. Similarly, FIG. 1 (E-1) illustrates an image obtained bysubtracting the image of FIG. 1 (A-1) from the image of FIG. 1 (B-1)with respect to the pair of parallax images. FIG. 1 (E-2) illustrates aluminance cross section along the horizontal direction at the vicinityof the center in the vertical direction. In this case, for simplifyingthe processing, when the difference value indicates a negative value,processing of truncating the negative value to be replaced with zero isperformed. As a result, all values in the difference image illustratedin FIG. 1 (E-1) indicate zero.

FIG. 1 (F-1) is an image obtained by adding and synthesizing the imageof FIG. 1(D-1) and FIG. 1 (E-1). Accordingly, a synthesized differenceimage illustrated in FIG. 1 (F-1) corresponds to an image obtained byremoving the object and the background from the image of FIG. 1 (C-1),and it indicates only an unnecessary component contained in the image ofFIG. 1 (C-1). As described above, by performing the differencecalculation with respect to each parallax image, only the unnecessarycomponent is maintained (that is, separated or extracted) and heunnecessary component can be determined.

In this embodiment, in order to calculate the image of FIG. 1 (F-1), asdescribed above, the difference calculations are performed twice beforethe addition synthesis processing, and alternatively, a calculation ofobtaining an absolute value of the difference may be performed to obtainan equivalent result as represented by expression (1) below.

Fig1F1(x,y)=|Fig1A1(x,y)−Fig1B1(x,y)|  (1)

In expression (1), symbols Fig1F1(x,y), Fig1A1(x,y), and Fig1B1(x,y)represent luminance values at each coordinate in the image of FIG. 1(F-1), FIG. 1 (A-1), and FIG. 1 (B-1), respectively. As a result, theresult (image) of FIG. 1 (F-1) can be obtained by a single calculation.

FIG. 1 (F-1) is an image (first unnecessary component image) relating tothe determined unnecessary component. While the “first unnecessarycomponent” is created by adding and synthesizing the difference imagesto be one image as illustrated in FIG. 1 (F-1) for simplifyingdescriptions, the difference images are separated as an “unnecessarycomponent image (1-1)” and an “unnecessary component image (1-2)”,respectively, so that next calculation processing is to be performedseparately. Processing described below is to be performed based on thedetermined unnecessary component. In this embodiment, the firstunnecessary component is not necessarily stored as a so-called “image”that is to be displayed later so that a user can view it. The firstunnecessary component image only needs to be usable as numerical dataduring the processing flow.

Next, correction processing of removing or reducing the unnecessarycomponent determined as described above is performed on an image to beoutput. If the correction processing of removing or reducing theunnecessary component is performed without considering a “gaindistribution” described below, FIG. 1 (F-1) as the first unnecessarycomponent image may be subtracted simply from the image of FIG. 1 (C-1).FIG. 1 (G-1) illustrates an unnecessary component reduction imageobtained by subtracting the components of FIG. 1 (F-1) from FIG. 1 (C-1)without considering the “gain distribution”. As a result, an image inwhich the unnecessary component has been reduced compared to a capturedimage generated by the “imaging without pupil division” is obtained asillustrated in FIG. 1 (C-1). As can be seen in this example, however,when an optical path of the ghost is not completely separated into theregions P1 and P2 and the light beam passes through both the regions P1and P2 at a certain ratio, the unnecessary component is not completelyremoved and accordingly, as illustrated in FIG. 1 (G-1), the unnecessarycomponent remains in the unnecessary component reduction image.

When one unnecessary component only remain, reduction processing may beperformed after a gain is applied uniformly to the first unnecessarycomponent image (FIG. 1 (F-1)) until the unnecessary component can besufficiently removed. For example, a case in which only the unnecessarycomponent A contained in FIG. 1 (C-1) is to be sufficiently removed willbe considered. The luminance value of the unnecessary component A is240, and it is greater by 100 than 140 as the luminance value of thebackground. Since the luminance value of the unnecessary component Acontained in the current first unnecessary component image (FIG. 1(F-1)) is 20, the gain-up may be performed (specifically, generally fivetimes of the luminance values of the first unnecessary component image)and then the first unnecessary component image of FIG. 1 (F-1) may besubtracted from the image of FIG. 1 (C-1).

As illustrated in FIG. 1, however, when a plurality of unnecessarycomponents are contained and a reduction rate is different for eachunnecessary component as illustrated in a graph of FIG. 1 (G-2), theunnecessary components cannot be removed by the simple gain-upprocessing. The reduction rate R(x,y) means a value calculated byexpression (2) below for each pixel. In expression (2), symbols G1(x,y)and C1(x,y) represent luminance values of FIG. 1 (G-1) and FIG. 1 (C-1)at each coordinate, respectively. When a denominator is zero, thereduction rate at the coordinate is zero.

R(x,y)=1−{G1(x,y)/C1(x,y)}  (2)

While the unnecessary component is illustrated as a square with auniform luminance value for simplicity in this embodiment, in reality,the reduction rate is different for each pixel. The reduction rateillustrated in FIG. 1 (G-2) is indicated by using percentage (%)corresponding to a value obtained by multiplying the result ofexpression (2) by 100. FIG. 1 (G-2) means that the unnecessary componentA is reduced by 8.3% compared to a captured image generated by the“imaging without pupil division” and the unnecessary component B isreduced by 18.2% compared to the captured image.

While only the unnecessary component A is described thus far,hereinafter, the unnecessary component B is also targeted by using thesame calculation. FIG. 8 is an example of output images obtained by theimage processing method in this embodiment, and it illustrates imagesrelating to the unnecessary component B. FIG. 8 (A-1) is a result ofmultiplying the luminance values of the first unnecessary componentimage entirely five times in order to remove the unnecessary component Asufficiently as described above. FIG. 8 (A-2) is a luminance crosssection along the horizontal direction at the vicinity of the center inthe vertical direction. FIG. 8 (B-1) is a result of subtracting theimage of FIG. 8 (A-1) from the image of FIG. 1 (C-1), and FIG. 8 (B-2)is a luminance cross section along the horizontal direction at thevicinity of the center in the vertical direction.

When the image of FIG. 8 (B-1) is seen, the unnecessary component B isovercorrected and the dark level depression occurs while a region inwhich the unnecessary component A originally existed and has beenremoved is buried by the background luminance. Thus, when the pluralityof unnecessary components exist and the reduction rates are differentfor each unnecessary component, the unnecessary components cannot beremoved by the simple gain-up processing. When the plurality ofunnecessary components with different reduction rates from each otherexist in an image, an application of a uniform gain in accordance with aluminance value of any one of the unnecessary components results inovercorrection or shortage of correction for other unnecessarycomponents. Accordingly, a good reduction result for all the unnecessarycomponents cannot be obtained by the uniform gain adjustment.

In order to solve the problem, this embodiment creates a gaindistribution in an image and changes a gain in the image based on thecreated gain distribution, instead of adjusting a gain uniformly in theimage. Accordingly, a plurality of unnecessary components with differentreduction rates can be effectively reduced.

In this embodiment, while the reduction rate is calculated by using aratio, as a method of further simply calculating the reduction rate, theprocessing of obtaining the difference between the luminance values ofFIG. 1 (G-1) and FIG. 1 (C-1) may be only performed. To be exact, it isdifferent from the ratio calculation in a behavior occurring when a gainis applied, but similarly, there is a tendency that the black leveldepression of the unnecessary component (unnecessary light component)after the reduction processing can be suppressed.

Next, as an example of a method of creating the gain distribution, amethod of creating the gain distribution based on the first unnecessarycomponent image will be described. Steps in the method until the firstunnecessary component image (FIG. 1 (F-1)) is calculated from theparallax images are the same as those in the method described above.Subsequently, with respect to a luminance value L (x,y) of the firstunnecessary component image at each coordinate, as represented byexpression (3) below, a gain distribution gain (x,y) at atwo-dimensional coordinate (x,y) is calculated. In expression (3),symbols α and β are parameters.

$\begin{matrix}{{{gain}\left( {x,y} \right)} = {\alpha \times \left( \frac{1}{L\left( {x,y} \right)} \right)^{\beta}}} & (3)\end{matrix}$

The concept of creating the gain distribution is to prevent theoccurrence of the dark level depression caused by the reduction of theunnecessary component with a high reduction rate too much due to thegain applied to effectively reduce the unnecessary component with a lowreduction rate. Accordingly, it is preferred that the gain applied tothe unnecessary component with the high reduction rate is suppressed tobe lower than that of each of the other unnecessary components. Since apart in which the luminance value is relatively high in the firstunnecessary component image is subtracted from the parallax image by avalue greater than that in another part, it can be estimated as a partwith a higher reduction rate. Accordingly, as represented by expression(3), the gain is set to be relatively low in the part with the highluminance value in the first unnecessary component image, and on theother hand the gain is set to be relatively high in the part with thelow luminance in the first unnecessary component image.

FIG. 9 (A-1) illustrates the gain distribution in this embodiment. Inthis case, parameters α and β are as follows.

α=262.324

β=1.322

In this embodiment, a method of calculating the parameters α and β isnot specifically limited. Even in the same unnecessary component, thebrightness (luminance) of the unnecessary component varies depending ona light source or an image capturing condition. A reduction rate of theunnecessary component also changes depending on each lens, andaccordingly to be exact, the values of the parameters α and β changedepending on a condition at the time of capturing an image. Accordingly,they may be automatically obtained by adaptive processing by using aconventional method, or alternatively a value of 1 as an initial valuemay be assigned to each of the parameters α and β so that a user canadjust an image by inputting a numerical value or using a slider duringthe reduction processing while seeing a degree of the reduction of theunnecessary component (unnecessary light).

Subsequently, as represented by expression (4) below, the gaindistribution gain (x,y) is multiplied by the luminance value L(x,y) ofthe first unnecessary component image to determine the unnecessarycomponent intensity I(x,y). By imaging the unnecessary componentintensity, a second unnecessary component image is obtained.

I(x,y)=gain(x,y)×L(x,y)  (4)

FIG. 9 (B-1) illustrates the second unnecessary component imagecalculated by expression (4). FIG. 9 (B-2) illustrates a luminance crosssection along the horizontal direction at the vicinity of the center inthe vertical direction. Since the concept of creating the gaindistribution is as described above, the method of creating the gaindistribution is not limited to expression (2) if it matches the concept.FIG. 9 (C-1) illustrates an image obtained by subtracting the secondunnecessary component image (FIG. 9 (B-1)) from the image of FIG. 1(C-1). FIG. (C-2) illustrates a luminance cross section along thehorizontal direction at the vicinity of the center in the verticaldirection. As illustrated in FIG. 9 (C-1) and FIG. (C-2), both ofunnecessary components A and B are effectively reduced.

In this embodiment, for the convenience of explanation, while the imageas a second unnecessary component image is created, in actualprocessing, the second unnecessary component image does not need to becreated and stored as a so-called “image” that can be displayed for auser later. The second unnecessary component image may be numerical datawhich can be used during the processing, and accordingly the unnecessarycomponent intensity may be determined based on the first unnecessarycomponent image and the gain distribution and a reduction image may becreated by performing reduction processing.

Next, referring to FIG. 10, the image processing method (determinationprocessing of the unnecessary component and reduction processing of theunnecessary component) in this embodiment will be described. FIG. 10 isa flowchart of illustrating the image processing method. Each step inFIG. 10 is performed by the system controller 210 or the image processor204 according to an image processing program as a computer program.

First, at step S101, the system controller 210 controls an image pickupdevice that is constituted by the optical system 201 and the imagepickup element 202 to photograph an object (capture an object image).The image processor 204 acquires a captured image as an input image.

Subsequently, at step S102, the image processor 204 generates a pair ofparallax images by using digital signals which are output from the imagepickup element 202 (pixel groups G1 and G2) and are obtained by the A/Dconversion of the A/D converter 203. In this embodiment, the imageprocessor 204 may perform typical development processing and variouskinds of image correction processing to generate the parallax images.

Subsequently, at step S103, the image processor 204 (unnecessarycomponent detector 204 a) obtains difference information of the pair ofparallax images. In other words, the image processor 204 generates adifference image of FIG. 1 (D-1) obtained by subtracting the image ofFIG. 1 (B-1) from the image of FIG. 1 (A-1) and a difference image ofFIG. 1 (E-1) obtained by subtracting the image of FIG. 1 (A-1) from theimage of FIG. 1 (B-1). In such a simple difference calculation,difference values of the unnecessary components indicate positive valuesor negative values. For example, in this embodiment, when the image ofFIG. 1 (B-1) is subtracted from the image of FIG. 1 (A-1) to generatethe difference image of FIG. 1 (D-1), luminance values of theunnecessary component contained in FIG. 1 (A-1) are larger thanluminance values of the unnecessary component contained in FIG. 1 (B-1).Accordingly, the difference values indicate positive values. Similarly,when the image of FIG. 1 (A-1) is subtracted from the image of FIG. 1(B-1), the difference values indicate negative values. In thisembodiment, for simplifying the unnecessary component reductionprocessing described below, processing of truncating the negative valuesto be zero is performed. Accordingly, all luminance values of an imageof FIG. 1 (E-1) indicate zero.

When difference information is obtained with respect to an imageincluding a close-range object, processing to align the positions of thepair of parallax images may be performed in order to remove an objectparallax component while a display of the parallax component is omittedin this embodiment. The alignment can be performed by determining ashift position at which a correlation between the pair of parallaximages is maximized while shifting a position of one of the parallaximages relative to a position of the other of the parallax images.Alternatively, the alignment may be performed by determining the shiftposition at which the sum of squares of the difference between theparallax images is minimized. An in-focus area in the parallax image maybe used to determine the shift position for the alignment.

An edge detection may be previously performed in each of the parallaximages to determine the shift position for the alignment using an imagecontaining the detected edge. According to this method, an edge with ahigh contrast is detected in the in-focus area, and on the other hand anout-of-focus area such as a background has a low contrast and is noteasily detected as an edge, and thus the shift position is inevitablydetermined with the in-focus area being emphasized. Furthermore, a stepof performing threshold processing or the like in order to remove theinfluence of a noise or the like may be added.

Subsequently, at step S104, the image processor 204 (unnecessarycomponent detector 204 a) determines a component remaining in thedifference image obtained at step S103 as an unnecessary component. Animage corresponding to the unnecessary component is referred to as afirst unnecessary component image. Specifically, by adding andsynthesizing an image of FIG. 1 (D-1) and an image of (FIG. 1 (E-1)),only a difference value of the unnecessary components contained in theimages of FIG. 1 (A-1) and FIG. 1 (B-1) is detected as positive values.The unnecessary component detector 204 a determines this as anunnecessary component, and it generates the first unnecessary componentimage (FIG. 1 (F-1)). However, the first unnecessary component image isnot necessarily generated or stored as described above, and accordinglythe difference image obtained at step S103 may be treated as anunnecessary component to improve a processing speed. In this case, stepS104 is skipped and the flow proceeds to step S105 subsequent to stepS103.

Subsequently, at step S105, the image processor 204 (gain distributionacquirer 204 b) generates (determines) a gain distribution (for example,FIG. 9 (A-1)) based on the unnecessary component obtained at step S104.

Subsequently, at step S106, the image processor 204 (unnecessarycomponent intensity determiner 204 c) determines an unnecessarycomponent intensity based on the unnecessary component determined atstep S103 or step S104 and the gain distribution determined at stepS105. Specifically, the unnecessary component intensity determiner 204 cmultiplies each luminance value of the first unnecessary component imageby the gain distribution at each coordinate as represented by expression(4) to determine the unnecessary component intensity.

Subsequently, at step S107, the image processor 204 generates a secondunnecessary component image (for example, an image of FIG. 9 (B-1))based on the result at step S106. Since the unnecessary componentintensity is determined by multiplying the difference image by the gaindistribution, in this embodiment, the unnecessary component reductionprocessing may be performed without step S107. In this case, step S107is skipped and step S108 is performed immediately after step S106.

Subsequently, at step S108, the image processor 204 generates an image(synthesized parallax image) which is equivalent to a captured imagegenerated by “imaging without pupil division” according to additionsynthesis processing of the parallax images. For example, by performingthe processing to add the parallax image of FIG. 1 (A-1) generated atstep S102 to the parallax image of FIG. 1 (B-1), the synthesizedparallax image, illustrated in FIG. 1 (C-1), on which the additionsynthesis processing has been performed is generated. Alternatively, atstep S108, the synthesized parallax image may be generated by adding thedigital signals which are output from the image pickup element 202(pixel groups G1 and G2) and are obtained by the A/D conversion of theA/D converter 203 without the step (step S102) at which the parallaximages are generated. Step S108 is not necessarily performed at thisposition, and the position at which this step is performed is notspecifically limited if it is performed before step S109 so that thesynthesized parallax image can be used at the next step S109.

Subsequently, at step S109, the image processor 204 (unnecessarycomponent reducer 204 d) performs correction processing to reduce orremove the unnecessary component from the synthesized parallax imagegenerated at step S108. Specifically, the unnecessary component reducer204 d subtracts the image of FIG. 9 (B-1) from the image of FIG. 1(C-1), and thus the unnecessary component can be reduced or removed.When the image of FIG. 9 (B-1) is not generated, the calculation of FIG.1 (C-1)−{FIG. (F-1)×FIG. 9(A-1)} may be directed performed. As a result,an unnecessary component reduction image is generated.

Finally, at step S110, the system controller 210 stores the output imagein which the unnecessary component has been removed or reduced, i.e.,unnecessary component reduction image (FIG. 9 (C-1)), in the imagerecording medium 209 or displays it on the display unit 205.

According to this embodiment, the unnecessary component which is formedby the unnecessary light (such as a ghost and a flare) from thedifference image based on the plurality of parallax images obtained by asingle image capturing (imaging) can be determined. In other words, theunnecessary component contained in a captured image can be determinedwithout image capturing a plurality of times. Furthermore, even when aplurality of unnecessary components (such as ghosts or flares) that passthrough pupil regions different from each other exist, the plurality ofunnecessary components can be effectively reduced by determining theunnecessary component intensity based on the gain distribution. In thisembodiment, for simplifying descriptions, an example of a gray scaleimage is described, and similarly it can be applied to a color image. Inthis case, the processing described above may be performed for eachcolor channel independently, and finally each color may be synthesizedto make one image.

Embodiment 2

Next, Embodiment 2 of the present invention will be described. Thisembodiment is different from Embodiment 1 with respect to the method ofcalculating the gain distribution. In this embodiment, a basicconfiguration of an image pickup apparatus is the same as that of theimage pickup apparatus 200 of Embodiment 1 described referring to FIG.5, and accordingly descriptions thereof will be omitted. An imageprocessing method of this embodiment is different from the imageprocessing method of Embodiment 1 only in the processing flow and thecalculation method and the result of the image processing method is thesame as that of Embodiment 1, and accordingly this embodiment will bedescribed referring to FIG. 1 or FIG. 9.

Next, referring to FIG. 11, the image processing method (determinationprocessing of the unnecessary component and reduction processing of theunnecessary component) in this embodiment will be described. FIG. 11 isa flowchart of illustrating the image processing method. Each step inFIG. 11 is performed by the system controller 210 or the image processor204 according to an image processing program as a computer program.

In FIG. 11, steps S201 to S204 are the same as steps S101 to S104 ofFIG. 10 in Embodiment 1, respectively. Subsequently, at step S205, theimage processor 204 performs addition synthesis processing on theparallax images to generate an image (synthesized parallax image) whichis equivalent to a captured image generated by “imaging without pupildivision”. In this embodiment, processing of adding the parallax imagesof FIG. 1 (A-1) and FIG. 1 (B-1) that are generated at step S202 isperformed, and accordingly the synthesized image (synthesized parallaximage) on which the addition synthesis processing has been performed isgenerated as illustrated in FIG. 1 (C-1). Alternatively, the imageprocessor 204 may add digital signals which are output from the imagepickup element 202 (pixel groups G1 and G2) and are obtained by the A/Dconversion of the A/D converter 203 without step S205 to generate thesynthesized parallax image.

Subsequently, at step S206, the image processor 204 generates a firstunnecessary component reduction image based on the synthesized parallaximage generated at step S205 and the unnecessary component (imaging ofthe unnecessary component corresponds to the first unnecessary componentimage) determined at step S204. Specifically, the image processor 204subtracts the image of FIG. 1 (F-1) from the image of FIG. 1 (C-1) toacquire the first unnecessary component reduction image (FIG. 1 (G-1)).

Subsequently, at step S207, the image processor 204 creates (calculates)the reduction rate distribution based on the synthesized parallax imagegenerated at step S205 and the first unnecessary component reductionimage generated at step S206. Specifically, the image processor 204calculates the reduction rate represented by expression (2) described inEmbodiment 1 for all pixels. When a denominator is zero, the reductionrate at the coordinate is zero. When a region in which the unnecessarycomponent exists is known, the calculation may be performed only in theregion in which the unnecessary component exists instead of performingthe calculation for all the pixels. In this case, subsequent processingmay be performed only in the region in which the unnecessary componentexists, or all the reduction rates may be set to zero in regions otherthan the region in which the unnecessary component exists. As describedabove, as a method of further simply calculating the reduction rate, theprocessing of obtaining the difference between the luminance values ofFIG. 1 (G-1) and FIG. 1 (C-1) may be only performed. To be exact, it isdifferent from the ratio calculation in a behavior occurring when a gainis applied, but similarly, there is a tendency that the black leveldepression of the unnecessary component (unnecessary light component)after the reduction processing can be suppressed. FIG. 12 is thereduction rate distribution which is obtained by using FIG. (C-1) andFIG. 1 (G-1) according to expression (2).

Subsequently, at step S208, the image processor 204 (gain distributionacquirer 204 b) calculates the gain distribution based on the reductionrate distribution obtained at step S207. Specifically, the gaindistribution acquirer 204 b calculates the gain distribution gain(x,y)based on the reduction rate distribution R(x,y) for each coordinate(x,y) as represented by expression (5) below. In expression (5), symbolsα and β are parameters.

$\begin{matrix}{{{gain}\left( {x,y} \right)} = {\alpha \times \left( \frac{1}{R\left( {x,y} \right)} \right)^{\beta}}} & (5)\end{matrix}$

FIG. 9 (A-1) illustrates the gain distribution in this embodiment.Parameters α and β in this case are as follows.

α=0.270

β=1.174

In this embodiment, a method of calculating the parameters α and β isnot specifically limited. Even in the same unnecessary component, thebrightness (luminance) of the unnecessary component varies depending onan image capturing condition such as a light source. A reduction rate ofthe unnecessary component also changes depending on each lens. Thus, tobe exact, the values of the parameters α and β change depending on acondition at the time of capturing an image. Accordingly, they may beautomatically obtained by adaptive processing by using a conventionalmethod, or alternatively a value of 1 as an initial value may beassigned to each of the parameters α and β so that a user can adjust animage by inputting a numerical value or using a slider during thereduction processing while seeing a degree of the reduction of theunnecessary component. Since the concept of creating the gaindistribution is as described in Embodiment 1, the method of creating thegain distribution is not limited to expression (5) if it matches theconcept.

Subsequently, at step S209, the image processor 204 (unnecessarycomponent intensity determiner 204 c) determines an unnecessarycomponent intensity based on the unnecessary component determined atstep S203 or step S204 and the gain distribution determined at stepS208. Specifically, the unnecessary component intensity determiner 204 cmultiplies each luminance value of the first unnecessary component imageby the gain distribution at each coordinate as represented by expression(4) to determine the unnecessary component intensity.

Subsequently, at step S210, the image processor 204 generates a secondunnecessary component image (for example, FIG. 9 (B-1)) based on theresult at step S209. Since the unnecessary component intensity isdetermined by multiplying the difference image by the gain distribution,in this embodiment, the unnecessary component reduction processing maybe performed without step S210. In this case, step S210 is skipped andstep S211 is performed immediately after step S209.

Subsequently, at step S211, the image processor 204 (unnecessarycomponent reducer 204 d) performs correction processing (generationprocessing of the second unnecessary component reduction image) toreduce or remove the unnecessary component from the synthesized parallaximage generated at step S205. Specifically, the unnecessary componentreducer 204 d subtracts the image of FIG. 9 (B-1) from the image of FIG.1 (C-1), and thus the unnecessary component can be reduced or removed.When the image of FIG. 9 (B-1) is not generated, the calculation of FIG.1 (C-1)−{FIG. 1 (F-1)×FIG. 9(A-1)} may be directed performed. As aresult, the second unnecessary component reduction image is generated.

Finally, at step S212, the system controller 210 stores the output imagein which the unnecessary component has been removed or reduced, i.e.,second unnecessary component reduction image (FIG. 9 (C-1)), in theimage recording medium 209 or displays it on the display unit 205.

According to this embodiment, the unnecessary component which is formedby the unnecessary light (such as a ghost and a flare) from thedifference image based on the plurality of parallax images obtained by asingle image capturing can be determined. In other words, theunnecessary component contained in a captured image can be determinedwithout image capturing a plurality of times. Furthermore, even when aplurality of unnecessary components (such as ghosts or flares) that passthrough pupil regions different from each other exist, the plurality ofunnecessary components can be effectively reduced by determining theunnecessary component intensity based on the gain distribution. In thisembodiment, for simplifying descriptions, an example of a gray scaleimage is described, and similarly it can be applied to a color image. Inthis case, the processing described above may be performed for eachcolor channel independently, and finally each color may be synthesizedto make one image.

Embodiment 3

Next, Embodiment 3 (multiple pupil division) of the present inventionwill be described. This embodiment is different from each of Embodiments1 and 2 in the number of parallaxes, the concept of parallax images, andthe expression of calculating a gain distribution. In this embodiment,the basic configuration of an image pickup apparatus and the basic flowof an image processing method are the same as those in Embodiment 2, andaccordingly descriptions thereof will be omitted.

FIG. 13 is a diagram of illustrating an image pickup element(light-receiving portion) in this embodiment. In FIG. 13, symbol MLrepresents a micro lens, and symbols G1, G2, G3, and G4 representlight-receiving portions (pixels), and one pixel G1, one pixel G2, onepixel G3, and one pixel G4 make a set. The image pickup element includesan array of a plurality of sets of the pixels G1, G2, G3, and G4. Theset of pixels G1, G2, G3, and G4 have a conjugate relation with the exitpupil EXP via the shared (that is, provided for each pixel set) microlens ML. In this embodiment, when an image which is equivalent to acaptured image generated by “imaging without pupil division” is output,averaging processing (addition averaging processing) on signals obtainedby the set of four pixels G1, G2, G3, and G4 are performed to generateone signal value.

In this embodiment, an example of a specific configuration of an opticalsystem is also the same as that of the optical system 201 in Embodiment1 described referring to FIGS. 6A to 6C, and accordingly descriptionsthereof will be omitted. However, while the two unnecessary components Aand B are contained as unnecessary components, an unnecessary componentC, which is not illustrated in FIGS. 6A to 6C, is further contained inthis embodiment.

FIG. 14 illustrates the regions P1, P2, P3, and P4 (pupil regions orpupil division regions) of the aperture stop STP, through which lightbeams incident on the pixels G1, G2, G3, and G4 illustrated in FIG. 13pass. The aperture stop STP can be assumed to correspond to the exitpupil EXP (i.e. virtual image when seen from an image plane position ofthe optical system 201) of the optical system 201, but in practice, itis often the case that the aperture stop STP and the exit pupil EXP aredifferent from each other. When a light beam from the high luminanceobject (SUN) passes through the aperture stop STP to be incident on eachpixel, it is divided into the regions P1, P2, P3, and P4 (pupilregions).

Next, referring to FIGS. 15 and 16, a method of determining anunnecessary component as an image component that appears through aphotoelectric conversion of unnecessary light in a captured imagegenerated by the image pickup apparatus 200 will be described. FIGS. 15and 16 are diagrams of illustrating a procedure of the image processingmethod in this embodiment.

FIG. 15 (A-1), FIG. 15 (B-1), FIG. 15 (C-1), and FIG. 15 (D-1)illustrate a set of parallax images which are obtained as a result ofthe photoelectric conversion of the light beams passing through theregions P1, P2, P3, and P4 (pupil regions) by the pixel groups G1, G2,G3, and G4, respectively. The set of parallax images contain theunnecessary components A, B, and C schematically illustrated as squares,which partially overlap with each other. Each of the unnecessarycomponents of the parallax images are located at the same position inthe parallax images of FIG. 15 (A-1), FIG. 15 (B-1), FIG. 15 (C-1), andFIG. 15 (D-1), and has a luminance different from each other. FIG. 15(A-2), FIG. 15 (B-2), FIG. 15 (C-2), and FIG. 15 (D-2) illustrateluminance cross sections of the respective parallax images along thehorizontal direction at the vicinity of the center in the verticaldirection. A numerical value in a graph of each drawing indicates aluminance value of an unnecessary component. For example, in FIG. 15(A-2), a luminance value at the background is 50, and luminance valuesof the unnecessary components A, B, and C are 180, 130, and 110,respectively. Apart where the unnecessary component overlaps indicates avalue obtained by adding and synthesizing the luminance value of each ofthe overlapped unnecessary component and the luminance value of thebackground.

FIG. 15 (E-1) is an image obtained by performing the averagingprocessing (addition averaging processing) on the images of FIG. 15(A-1), FIG. 15 (B-1), FIG. 15 (C-1), and FIG. 15 (D-1). Specifically, aluminance value of FIG. (E-1) at each coordinate is calculated by addingluminance values of the images of FIG. 15 (A-1), FIG. 15 (B-1), FIG. 15(C-1), and FIG. 15 (D-1) at each coordinate and then dividing the addedluminance value by four. This is equivalent to a captured imagegenerated by the “imaging without pupil division” in the image pickupapparatus of this embodiment. FIG. 15 (E-2) illustrates a luminancecross section of the image of FIG. 15 (E-1) along the horizontaldirection at the vicinity of the center in the vertical direction.

FIG. 16 (A-1), FIG. 16 (B-1), and FIG. 16 (C-1) are difference imagesobtained by subtracting the images of FIG. 15 (B-1), FIG. 15 (C-1), andFIG. 15 (D-1), respectively, from the image of FIG. 15 (A-1) as areference image with respect to the set of parallax images. Similarly toEmbodiment 1, these parallax images contain the unnecessary componentsas difference information. Furthermore, similarly to Embodiment 1, whilethere are parts in which the unnecessary components contained in FIG. 16(A-1), FIG. 16 (B-1), and FIG. 16 (C-1) are calculated as negativevalues by the difference calculation, for simplifying the unnecessarycomponent reduction processing at the subsequent stage, the negativevalues are truncated to be zero. The same is true for all the otherdifference images. FIG. 16 (D-1) is information (maximum differenceinformation or maximum difference image) obtained by extracting amaximum value between difference information at each pixel position inthe difference images of FIG. 16 (A-1), FIG. 16 (B-1), and FIG. 16 (C-1)as difference information acquired as two-dimensional data.

FIG. 16 (A-2), FIG. 16 (B-2), and FIG. 16 (C-2) are difference imagesobtained by subtracting the images of FIG. 15 (A-1), FIG. 15 (C-1), andFIG. 15 (D-1), respectively, from the image of FIG. 15 (B-1) as thereference image with respect to the set of parallax images. FIG. 16(D-2) is the maximum difference information between the differenceinformation at each pixel position in the difference images of FIG. 16(A-2), FIG. 16 (B-2), and FIG. (C-2) as difference information acquiredas two-dimensional data.

FIG. 16 (A-3), FIG. 16 (B-3), and FIG. 16 (C-3) are difference imagesobtained by subtracting the images of FIG. 15 (A-1), FIG. 15 (B-1), andFIG. 15 (D-1), respectively, from the image of FIG. 15 (C-1) as thereference image with respect to the set of parallax images. FIG. 16(D-3) is the maximum difference information between the differenceinformation at each pixel position in the difference images of FIG. 16(A-3), FIG. 16 (B-3), and FIG. (C-3) as difference information acquiredas two-dimensional data.

FIG. 16 (A-4), FIG. 16 (B-4), and FIG. 16 (C-4) are difference imagesobtained by subtracting the images of FIG. 15 (A-1), FIG. 15 (B-1), andFIG. 15 (C-1), respectively, from the image of FIG. 15 (D-1) as thereference image with respect to the set of parallax images. FIG. 16(D-4) is the maximum difference information between the differenceinformation at each pixel position in the difference images of FIG. 16(A-4), FIG. 16 (B-4), and FIG. (C-4) as difference information acquiredas two-dimensional data. The maximum difference information is a resultobtained by extracting the unnecessary component from each parallaximage.

Hereinafter, as described in Embodiments 1 and 2, the case in which theunnecessary component (imaging of the unnecessary component correspondsto the first unnecessary component image) is to determined isconsidered. In this case, as described above, the unnecessary componentis extracted as maximum difference information for each parallax image,and accordingly it is considered that each of pieces of maximumdifference information corresponds to the first unnecessary componentimage as a method. However, it is necessary to perform the subsequentprocessing as an image a number of times corresponding to the number ofthe parallax images, and thus the processing step is complicated. Inorder to solve the problem, in this embodiment, the subsequentprocessing is simplified by synthesizing each of pieces of the maximumdifference information to one. Specifically, addition averagingprocessing is performed on the images of FIG. 16 (D-1), FIG. 16 (D-2),FIG. 16 (D-3), and FIG. 16 (D-4), and then the images are synthesized(combined). FIG. 16 (E-1) is a result of synthesizing the images, andFIG. 16 (E-2) is a luminance cross section along the horizontaldirection at the vicinity of the center in the vertical direction. FIG.16 (F-1) is an image obtained by subtracting FIG. 16 (E-1) as the firstunnecessary component image from FIG. 15 (E-1) as the synthesizedparallax image, and it corresponds to the first unnecessary componentreduction image of FIG. 11 described in Embodiment 2. FIG. 16 (F-2) is aluminance cross section along the horizontal direction at the vicinityof the center in the vertical direction.

As described above, even when the number of parallaxes increases, “thesynthesized parallax images” and “the first unnecessary component image”can be calculated. The flow of subsequent processing and the basicconcept are the same as those of Embodiment 2, and accordingly,hereinafter, parts different from those of Embodiment 2 will be mainlydescribed according to the processing flow of FIG. 11.

This embodiment is different from Embodiment 2 in the method ofcalculating the gain distribution at step S208. FIG. 17 is a diagram ofillustrating the reduction rate distribution created (calculated) atstep S207.

In this embodiment, the gain distribution gain(x,y) is calculated byusing expression (6) below. In expression (6), symbol R(x,y) is thereduction rate distribution, and symbols A, B, C, D, and E areparameters.

$\begin{matrix}{{{gain}\left( {x,y} \right)} = {A + {B \times \left( \frac{1}{R\left( {x,y} \right)} \right)^{1}} + {C \times \left( \frac{1}{R\left( {x,y} \right)} \right)^{2}} + {D \times \left( \frac{1}{R\left( {x,y} \right)} \right)^{3}} + {E \times \left( \frac{1}{R\left( {x,y} \right)} \right)^{4}}}} & (6)\end{matrix}$

The reduction rate R(x,y) is a value which is calculated by expression(7) below for each pixel, similarly to the way of thinking by expression(2). When a denominator is zero, the reduction rate at the coordinate iszero.

R(x,y)=1−{F1(x,y)/E1(x,y)}  (7)

In expression (7), symbols F1(x,y) and E1(x,y) represent luminancevalues at each coordinate in FIG. 16 (F-1) and FIG. 15 (E-1),respectively.

FIG. 18 (A-1) illustrates the gain distribution in this embodiment.Parameters A, B, C, D, and E in this case are as follows.

A=75.69478

B=−88.7086

C=37.02509

D=−6.48479

E=0.408826

In this embodiment, a method of calculating the parameters A, B, C, D,and E is not specifically limited. Even in the same unnecessarycomponent, the brightness (luminance) of the unnecessary componentvaries depending on an image capturing condition such as a light source.A reduction rate of the unnecessary component also changes depending oneach lens. Thus, to be exact, the values of the parameters changedepending on a condition at the time of capturing an image. Accordingly,they may be automatically obtained by adaptive processing by using aconventional method, or alternatively arbitrary values as initial valuesmay be assigned to the respective parameters. The concept of creatingthe gain distribution is as described in Embodiments 1 and 2, and themethod of creating the gain distribution is not limited to expression(6) if it matches the concept.

Subsequently, at step S209, the image processor 204 (unnecessarycomponent intensity determiner 204 c) determines an unnecessarycomponent intensity based on the determined unnecessary component (firstunnecessary component image) and the gain distribution. Specifically,the unnecessary component intensity determiner 204 c multiplies theluminance value L(x,y) at each coordinate (x,y) in the first unnecessarycomponent image by the gain (x,y) as represented by expression (4) tocalculate the unnecessary component intensity I(x,y).

Subsequently, at step S210, the image processor 204 generates a secondunnecessary component image (FIG. 18 (B-1)) based on the result at stepS209. FIG. 18 (B-2) illustrates a luminance cross section along ahorizontal direction at the vicinity of the center in a verticaldirection of FIG. 18 (B-1). As described in Embodiments 1 and 2, stepsS209 and S210 may be skipped and step S211 may be performed immediatelyafter step S208.

Subsequently, at step S211, the image processor 204 (unnecessarycomponent reducer 204 d) performs correction processing (generationprocessing of the second unnecessary component reduction image) toreduce or remove the unnecessary component from the synthesized parallaximage generated at step S205. Specifically, the unnecessary componentreducer 204 d subtracts the image of FIG. 18 (B-1) from the image ofFIG. 15 (E-1), and thus the unnecessary component can be reduced orremoved. When the image of FIG. 18 (B-1) is not generated, thecalculation of FIG. 15 (E-1)−{FIG. 16 (E-1)×FIG. 18 (A-1)} may bedirected performed.

Finally, at step S212, the system controller 210 stores the output imagein which the unnecessary component has been removed or reduced, i.e.,second unnecessary component reduction image (FIG. 18 (C-1)), in theimage recording medium 209 or displays it on the display unit 205. FIG.18 (C-2) illustrates a luminance cross section along a horizontaldirection at the vicinity of the center in a vertical direction of FIG.18 (C-1).

According to this embodiment, the unnecessary component which is formedby the unnecessary light (such as a ghost and a flare) from thedifference image based on the plurality of parallax images obtained by asingle image capturing can be determined. In other words, theunnecessary component contained in a captured image can be determinedwithout image capturing a plurality of times. Furthermore, even when aplurality of unnecessary components (such as ghosts or flares) that passthrough pupil regions different from each other exist, the plurality ofunnecessary components can be effectively reduced by determining theunnecessary component intensity based on the gain distribution. In thisembodiment, for simplifying descriptions, an example of a gray scaleimage is described, and similarly it can be applied to a color image. Inthis case, the processing described above may be performed for eachcolor channel independently, and finally each color may be synthesizedto make one image.

Embodiment 4

Next, Embodiment 4 of the present invention will be described. Ren. Nget al., “Light Field Photography with a Hand-held Plenoptic Camera”(Stanford Tech Report CTSR 2005-2) discloses a “plenoptic camera”. The“plenoptic camera” can acquire information of the position and angle ofa light beam from an object by using a technique called “light fieldphotography”.

FIG. 19 illustrates an image pickup system of an image pickup apparatusin this embodiment, and illustrates a configuration of the image pickupsystem of the “plenoptic camera”. An optical system 301 (image pickupoptical system) includes a primary lens (image pickup lens) 301 b and anaperture stop 301 a. A micro lens array 301 c is disposed at an imagingposition of the optical system 301, and an image pickup element 302 isdisposed behind (closer to an image than) the micro lens array 301 c.The micro lens array 301 c has a function as a separator (separatingmember) that prevents a light beam passing through, for example, a pointA in an object space from being mixed with a light beam passing througha point near the point A on the image pickup element 302. FIG. 19illustrates that a top beam, a primary light beam, and a bottom beamfrom the point A are received by pixels different from each other. Thus,the light beams passing through the point A can be separately acquireddepending on their angles.

Todor Georgive et al., “Full Resolution Light Field Rendering” (AdobeTechnical Report January 2008) discloses configurations of an imagepickup system illustrated in FIGS. 20 and 21 that acquire information(light field) of the position and angle of a light beam.

With the configuration of the image pickup system illustrated in FIG.20, the micro lens array 301 c is disposed behind (closer to an imagethan) the imaging position of the primary lens 301 b to reimage thelight beams passing through the point A on the image pickup element 302,thereby separately acquiring the light beams depending on their angles.With the configuration of the image pickup system illustrated in FIG.21, the micro lens array 301 c is disposed in front of (closer to anobject than) the imaging position of the primary lens 301 b to image thelight beams passing through the point A on the image pickup element 302,thereby separately acquiring the light beams depending on their angles.In both configurations, light beams passing through a pupil of theoptical system 301 are separated depending on passed regions (passedpositions) in the pupil. In these configurations, the image pickupelement 302 may employ a conventional image pickup element including onemicro lens ML and one light-receiving portion G1 that are paired via acolor filter CF as illustrated in FIG. 22.

The optical system 301 illustrated in FIG. 19 yields an image asillustrated in FIG. 23A. FIG. 23B is an enlarged view of one of arrayedcircles in FIG. 23A. One circle represents the aperture stop STP, and aninside thereof is divided by a plurality of pixels Pj (j=1, 2, 3, . . .). This configuration allows the intensity distribution of the pupilwithin one circle to be acquired. The optical system 301 illustrated inFIGS. 20 and 21 are used to obtain parallax images illustrated in FIG.24. The parallax images as illustrated in FIG. 24 may be obtained byrearranging and reconstructing the pixels Pj in the circles (aperturestops STP) in an image illustrated in FIG. 23A.

As described in Embodiment 1 to 3, unnecessary light such as ghostpasses through the pupil with biased distribution across the pupil.Thus, the image pickup apparatus in this embodiment that performs imagepickup through divided regions of the pupil may employ the imageprocessing methods described in Embodiment 1 to 3 to determineunnecessary components and further reduce them.

In another example, parallax images are obtained by capturing images ofan identical object through a plurality of cameras as illustrated inFIG. 25. Thus, these cameras may employ the image processing methodsdescribed in Embodiment 1 to 3. C1, C2, and C3 represent separate imagepickup apparatuses, but they may be regarded as a single image pickupapparatus that performs image pickup through three divided regions of alarge pupil. Alternatively, as illustrated in FIG. 26, the pupildivision may be achieved by providing one image pickup apparatus with aplurality of optical systems OSj (j=1, 2, 3, . . . ).

Each of the embodiments describes the image pickup apparatus thatperforms the image processing method of each embodiment (is providedwith the image processing apparatus), but the image processing method ofeach embodiment may be performed by an image processing programinstalled in a personal computer. In this case, the personal computercorresponds to the image processing apparatus of each embodiment. Thepersonal computer takes in (acquires) an image (input image) generatedby the image pickup apparatus and yet to be provided with imageprocessing, and outputs an image obtained by performing the imageprocessing by the image processing program.

In each embodiment, the image processing apparatus (image processor 204)includes the generator (unnecessary component detector 204 a, ordeterminer) the gain distribution determiner (gain distribution acquirer204 b, or calculator), the intensity determiner (unnecessary componentintensity determiner 204 c), and the reducer (unnecessary componentreducer 204 d). The generator generates difference information relatingto a plurality of parallax images. The gain distribution determinerdetermines a gain distribution based on the difference information. Theintensity determiner determines an unnecessary component intensity basedon the gain distribution. The reducer generates an unnecessary componentreduction image in which an unnecessary component is reduced based onthe parallax image and the unnecessary component intensity.

Preferably, the reducer reduces the unnecessary component from asynthesized parallax image obtained by synthesizing the parallax imagesbased on the unnecessary component intensity to generate the unnecessarycomponent reduction image (S109, S211). Preferably, the generatordetermines an unnecessary component based on the difference information(S104, S204). Then, the gain distribution determiner determines the gaindistribution based on the unnecessary component. More preferably, thegenerator generates an image (first unnecessary component image)relating to the unnecessary component. Then, the gain distributiondeterminer determines the gain distribution (as represented byexpression (3)) depending on a luminance value L(x,y) of the image(first unnecessary component image). More preferably, the intensitydeterminer determines the unnecessary component intensity I(x,y) basedon the gain distribution and the luminance value of the image (asrepresented by expression (4)).

Preferably, the generator determines a reduction rate distributionR(x,y) based on the parallax image and the unnecessary component. Then,the gain distribution determiner determines the gain distributiondepending on the reduction rate distribution. More preferably, thegenerator determines the reduction rate distribution based on theparallax image and the difference information (S206, S207).

Preferably, the difference information is obtained by setting each ofthe plurality of parallax images as a reference image and calculating adifference between the reference image and a parallax image other thanthe reference image. Preferably, the difference information is obtainedby calculating an absolute value of a difference between two parallaximages. Preferably, the plurality of parallax images are imagesgenerated based on light beams passing through regions different fromeach other in a pupil of an optical system.

As a modification of each embodiment, the image processing apparatus(image processor 204) may include an unnecessary component determiner(unnecessary component detector 204 a), a gain distribution determiner(gain distribution acquirer 204 b), and a reducer (unnecessary componentreducer 204 d). The unnecessary component determiner generatesdifference information relating to a plurality of parallax images todetermine an unnecessary component based on the difference information.The gain distribution determiner determines a gain distribution based onthe unnecessary component. The reducer generates an unnecessarycomponent reduction image in which an unnecessary component is reducedbased on the parallax image, the unnecessary component, and the gaindistribution.

Other Embodiments

Embodiment (s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

According to each embodiment, an image processing apparatus, an imagepickup apparatus, an image processing method, and a non-transitorycomputer-readable storage medium which are capable of effectivelydetermining an intensity of an unnecessary component contained in acaptured image without imaging a plurality of times to reduce theunnecessary component from the captured image can be provided.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-039964, filed on Mar. 2, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: agenerator configured to generate difference information relating to aplurality of parallax images; a gain distribution determiner configuredto determine a gain distribution based on the difference information; anintensity determiner configured to determine an unnecessary componentintensity based on the gain distribution; and a reducer configured togenerate an unnecessary component reduction image in which anunnecessary component is reduced based on the parallax image and theunnecessary component intensity.
 2. The image processing apparatusaccording to claim 1, wherein the reducer is configured to reduce theunnecessary component from a synthesized parallax image obtained bysynthesizing the parallax images based on the unnecessary componentintensity to generate the unnecessary component reduction image.
 3. Theimage processing apparatus according to claim 1, wherein: the generatoris configured to determine an unnecessary component based on thedifference information, and the gain distribution determiner isconfigured to determine the gain distribution based on the unnecessarycomponent.
 4. The image processing apparatus according to claim 3,wherein: the generator is configured to generate an image relating tothe unnecessary component, and the gain distribution determiner isconfigured to determine the gain distribution depending on a luminancevalue of the image.
 5. The image processing apparatus according to claim4, wherein the intensity determiner is configured to determine theunnecessary component intensity based on the gain distribution and theluminance value of the image.
 6. The image processing apparatusaccording to claim 3, wherein: the generator is configured to determinea reduction rate distribution based on the parallax image and theunnecessary component, and the gain distribution determiner isconfigured to determine the gain distribution depending on the reductionrate distribution.
 7. The image processing apparatus according to claim6, wherein the generator is configured to determine the reduction ratedistribution based on the parallax image and the difference information.8. The image processing apparatus according to claim 1, wherein thedifference information is obtained by setting each of the plurality ofparallax images as a reference image and calculating a differencebetween the reference image and a parallax image other than thereference image.
 9. The image processing apparatus according to claim 1,wherein the difference information is obtained by calculating anabsolute value of a difference between two parallax images.
 10. Theimage processing apparatus according to claim 1, wherein the pluralityof parallax images are images generated based on light beams passingthrough regions different from each other in a pupil of an opticalsystem.
 11. An image pickup apparatus comprising: an image pickup deviceconfigured to photoelectrically convert an optical image formed via anoptical system to output a plurality of parallax images; a determinerconfigured to determine difference information relating to the pluralityof parallax images; a calculator configured to calculate a gaindistribution based on the difference information; an intensitydeterminer configured to determine an unnecessary component intensitybased on the gain distribution; and a reducer configured to generate anunnecessary component reduction image in which an unnecessary componentis reduced based on the parallax image and the unnecessary componentintensity.
 12. The image pickup apparatus according to claim 11,wherein: the plurality of parallax images are images generated based onlight beams passing through regions different from each other in a pupilof the optical system, the image pickup device includes a plurality ofpixels sharing a single micro lens, and the plurality of pixels areconfigured to receive the light beams passing through the regionsdifferent from each other in the pupil of the optical system.
 13. Theimage pickup apparatus according to claim 11, wherein the plurality ofparallax images are images generated by guiding light beams passingthrough regions different from each other in a pupil of the opticalsystem to pixels of the image pickup device different from each other.14. An image processing method comprising the steps of: determiningdifference information relating to a plurality of parallax images;calculating a gain distribution based on the difference information;determining an unnecessary component intensity based on the gaindistribution; and generating an unnecessary component reduction image inwhich an unnecessary component is reduced based on the parallax imageand the unnecessary component intensity.
 15. A non-transitorycomputer-readable storage medium which stores a program causing acomputer to execute a process comprising the steps of: determiningdifference information relating to a plurality of parallax images;calculating a gain distribution based on the difference information;determining an unnecessary component intensity based on the gaindistribution; and generating an unnecessary component reduction image inwhich an unnecessary component is reduced based on the parallax imageand the unnecessary component intensity.
 16. An image processingapparatus comprising: an unnecessary component determiner configured togenerate difference information relating to a plurality of parallaximages to determine an unnecessary component based on the differenceinformation; a gain distribution determiner configured to determine again distribution based on the unnecessary component; and a reducerconfigured to generate an unnecessary component reduction image in whichan unnecessary component is reduced based on the parallax image, theunnecessary component, and the gain distribution.
 17. An image pickupapparatus comprising: an image pickup device configured tophotoelectrically convert an optical image formed via an optical systemto output a plurality of parallax images; an unnecessary componentdeterminer configured to generate difference information relating to theplurality of parallax images to determine an unnecessary component basedon the difference information; a gain distribution determiner configuredto determine a gain distribution based on the unnecessary component; anda reducer configured to generate an unnecessary component reductionimage in which an unnecessary component is reduced based on the parallaximage, the unnecessary component, and the gain distribution.
 18. Animage processing method comprising the steps of: generating differenceinformation relating to a plurality of parallax images to determine anunnecessary component based on the difference information; determining again distribution based on the unnecessary component; and generating anunnecessary component reduction image in which an unnecessary componentis reduced based on the parallax image, the unnecessary component, andthe gain distribution.
 19. A non-transitory computer-readable storagemedium which stores a program causing a computer to execute a processcomprising the steps of: generating difference information relating to aplurality of parallax images to determine an unnecessary component basedon the difference information; determining a gain distribution based onthe unnecessary component; and generating an unnecessary componentreduction image in which an unnecessary component is reduced based onthe parallax image, the unnecessary component, and the gaindistribution.